FEAT Handle trial assertions through aggregate threshold evaluation#115
FEAT Handle trial assertions through aggregate threshold evaluation#115behnam-o wants to merge 1 commit into
Conversation
| if call.excinfo is None: | ||
| result = Result( | ||
| safe=True, | ||
| status=SafetyStatus.SAFE, | ||
| summary="Trial completed without recording a Result.", | ||
| ) | ||
| else: | ||
| summary = str(call.excinfo.value) or "Trial assertion did not pass." | ||
| result = Result( | ||
| safe=False, | ||
| status=SafetyStatus.UNDETERMINED, | ||
| summary=summary, | ||
| ) | ||
| collector.record(result=result) |
There was a problem hiding this comment.
this part makes me a bit uneasy - would also have to check if we are forwarding all the other metadata fields like category, etc.
|
|
||
| _record_missing_trial_result(collector=collector, call=call) | ||
| report.outcome = "passed" | ||
| report.longrepr = None | ||
| return report |
There was a problem hiding this comment.
I also am not sure about how we want to think about the population number total reported by pytest: if I have a suite of 10 tests, with differing number of trials, we would potentially be weighting those tests as it currently stands - perhaps more thought about hooking into execution loop to run our own loop and then report to pytest just 1 result (akin to pytest-loop)
|
this was opened to allow overriding pytest exit code when implementing trials + threshold in their current state. closing in favor of the work that will introduce an alternative way to achieve that |
2 participants
Trial clone assertion failures were reported as individual pytest failures, bypassing the trial group’s aggregate threshold.
This change intercepts assertion failures for
@pytest.mark.trialcalls, records missing outcomes asUNDETERMINED, and marks clones as passed so the aggregate threshold determines the final result. Successful trials without explicit results are recorded asSAFE. Non-trial assertions and non-assertion exceptions retain normal pytest behavior. Tests cover threshold outcomes, result deduplication, and failure preservation.